How do humans learn about the reliability of automation?

In a range of settings, human operators make decisions with the assistance of automation, the reliability of which can vary depending upon context. Currently, the processes by which humans track the level of reliability of automation are unclear. In the current study, we test cognitive models of learning that could potentially explain how humans track automation reliability. We fitted several alternative cognitive models to a series of participants' judgements of automation reliability observed in a maritime classification task in which participants were provided with automated advice. We examined three experiments including eight between-subjects conditions and 240 participants in total. Our results favoured a two-kernel delta-rule model of learning, which specifies that humans learn by prediction error, and respond according to a learning rate that is sensitive to environmental volatility. However, we found substantial heterogeneity in learning processes across participants. These outcomes speak to the learning processes underlying how humans estimate automation reliability and thus have implications for practice.

The role of episodic memory sampling in evaluation.

Many models of choice assume that people retrieve memories of past experiences and use them to guide evaluation and choice. In this paper, we examine whether samples of recalled past experiences do indeed underpin our evaluations of options. We showed participants sequences of numerical values and asked them to recall as many of those values as possible and also to state how much they would be willing to pay for another draw from the sequence. Using Bayesian mixed effects modeling, we predicted participants' evaluation of the sequences at the group level from either the average of the values they recalled or the average of the values they saw. Contrary to the predictions of recall-based models, people's evaluations appear to be sensitive to information beyond what was actually recalled. Moreover, we did not find consistent evidence that memory for specific items is sufficient to predict evaluation of sequences. We discuss the implications for sampling models of memory and decision-making and alternative explanations.

The Perception of Automation Reliability and Acceptance of Automated Advice.

OBJECTIVE: Examine (1) the extent to which humans can accurately estimate automation reliability and calibrate to changes in reliability, and how this is impacted by the recent accuracy of automation; and (2) factors that impact the acceptance of automated advice, including true automation reliability, reliability perception, and the difference between an operator's perception of automation reliability and perception of their own reliability. BACKGROUND: Existing evidence suggests humans can adapt to changes in automation reliability but generally underestimate reliability. Cognitive science indicates that humans heavily weight evidence from more recent experiences. METHOD: Participants monitored the behavior of maritime vessels (contacts) in order to classify them, and then received advice from automation regarding classification. Participants were assigned to either an initially high (90%) or low (60%) automation reliability condition. After some time, reliability switched to 75% in both conditions. RESULTS: Participants initially underestimated automation reliability. After the change in true reliability, estimates in both conditions moved towards the common true reliability, but did not reach it. There were recency effects, with lower future reliability estimates immediately following incorrect automation advice. With lower initial reliability, automation acceptance rates tracked true reliability more closely than perceived reliability. A positive difference between participant assessments of the reliability of automation and their own reliability predicted greater automation acceptance. CONCLUSION: Humans underestimate the reliability of automation, and we have demonstrated several critical factors that impact the perception of automation reliability and automation use. APPLICATION: The findings have potential implications for training and adaptive human-automation teaming.

Evidence Against Novelty-Gated Encoding in Serial Recall.

Novelty-gated encoding is the assumption that events are encoded more strongly into memory when they are more novel in comparison to previously encoded events. It is a core assumption of the SOB model of serial recall (Farrell & Lewandowsky, 2002). We present three experiments testing some predictions from novelty-gated encoding. Experiment 1 shows that the probability of recalling the third item in a list correctly does not depend on whether it is preceded by phonologically similar or dissimilar items. Experiment 2 shows that in lists of items from three classes (nonwords, spatial locations, and abstract drawings) the probability of recalling an item does not depend on whether it is preceded by items from the same or another class. Experiment 3 used a complex-span paradigm varying the phonological similarity of words that are read aloud as distractors in between memory items. Contrary to a prediction from novelty-gated encoding, similar distractors did not impair memory more than dissimilar distractors. The results question the assumption of novelty-gated encoding in serial recall. We discuss alternative explanations for the phenomena that this assumption has previously helped to explain. The present evidence against novelty-gated encoding might point to boundary conditions for the role of prediction error in the acquisition of memories.

Human behavioral response to fluctuating automation reliability.

Human perception of automation reliability and automation acceptance behaviours are key to effective human-automation teaming. This study examined factors that impact perceptions of automation reliability over time and the acceptance of automated advice. Participants completed a maritime vessel classification task in which they classified vessels (contacts) with the assistance of automation. In Experiment 1 automation reliability successively switched from high to low (or vice versa). In Experiment 2 automation reliability decreased by varying magnitudes before returning to high. Participants did not initially calibrate to true reliability and experiencing low automation reliability reduced future reliability estimates when experiencing subsequent high reliability. Automation acceptance was predicted by positive differences between participant perception of automation reliability and confidence in their own manual classification reliability. Experiencing low automation reliability caused perceptions of reliability and automation acceptance rates to diverge. These findings have important implications for training and adaptive human-automation teaming in complex work environments.

Examining the role of information integration in the continued influence effect using an event segmentation approach.

Misinformation regarding the cause of an event often continues to influence an individual's event-related reasoning, even after they have received a retraction. This is known as the continued influence effect (CIE). Dominant theoretical models of the CIE have suggested the effect arises primarily from failures to retrieve the correction. However, recent research has implicated information integration and memory updating processes in the CIE. As a behavioural test of integration, we applied an event segmentation approach to the CIE paradigm. Event segmentation theory suggests that incoming information is parsed into distinct events separated by event boundaries, which can have implications for memory. As such, when an individual encodes an event report that contains a retraction, the presence of event boundaries should impair retraction integration and memory updating, resulting in an enhanced CIE. Experiments 1 and 2 employed spatial event segmentation boundaries in an attempt to manipulate the ease with which a retraction can be integrated into a participant's mental event model. While Experiment 1 showed no impact of an event boundary, Experiment 2 yielded evidence that an event boundary resulted in a reduced CIE. To the extent that this finding reflects enhanced retrieval of the retraction relative to the misinformation, it is more in line with retrieval accounts of the CIE.

Durability of treatment effects following internet-delivered cognitive behavioural therapy for depression and anxiety delivered within a routine care setting.

OBJECTIVE: To investigate post-treatment relapse and remission rates 3, 6 and 9 months after completion of an acute phase of a clinician-supported internet-delivered cognitive-behavioural therapy (iCBT) for anxiety and depressive symptoms, within a routine care setting. METHOD: Secondary analysis from a 12-month pragmatic randomized-controlled trial delivered within the Improving Access to Psychological Therapies (IAPT) programme in England. Participants in the intervention arm were included if they met criteria for reliable recovery from depression (PHQ-9) and anxiety (GAD-7) at post-treatment assessment. Survival analysis was used to assess durability of treatment effects and determine predictors to relapse at 3-, 6- and 9-month follow-up. Hazard ratios predicting time-to-relapse were estimated with semi-parametric Cox proportional hazards model. RESULTS: Of the 241 participants in the intervention arm, 89 participants met the criteria for reliable recovery from depression and anxiety at the post-treatment assessment. Of these 89 eligible cases, 29.2% relapsed within the 9-month period, with 70.8% remaining in remission at 9 months post-treatment. Of those who relapsed, 53.8% experienced a relapse of depression and anxiety; 7.7% experienced a relapse of depression only; and 38.4% experienced a relapse of anxiety only. Younger age, having a long-term condition, and residual symptoms of anxiety at end-of-treatment were all significant predictors of relapse. CONCLUSIONS: This study is the first to explore the remission and relapse rates after an acute phase of iCBT treatment, within a routine, stepped-care setting. The results add to the scarce literature on the durability of the effects of iCBT treatment in routine care settings, where patients are not typically followed up after receiving a completed course of treatment.

The influence of three-dimensional cues on body size judgements.

Research has shown that body size judgements are frequently biased, or inaccurate. Critically, judgement biases are further exaggerated for individuals with eating disorders, a finding that has been attributed to difficulties integrating body features into a perceptual whole. However, current understanding of which body features are integrated when judging body size is lacking. In this study, we examine whether individuals integrate three-dimensional (3D) cues to body volume when making body size judgements. Computer-generated body stimuli were presented in a 3D Virtual Reality environment. Participants (N = 412) were randomly assigned to one of the two conditions: in one condition, the to-be-judged body was displayed binocularly (containing 3D cues to body volume); in the other, bodies were presented monocularly (two-dimensional [2D] cues only). Across 150 trials, participants were required to make a body size judgement of a target female body from a third-person point of view using an unmarked visual analogue scale (VAS). It was found that 3D cues significantly influenced body size judgements. Namely, thin 3D bodies were judged smaller, and overweight 3D bodies were judged larger, than their 2D counterpart. Furthermore, to reconcile these effects, we present evidence that the two perceptual biases, regression to the mean and serial dependence, were reduced by the additional 3D feature information. Our findings increase our understanding of how body size is perceptually encoded and creates testable predictions for clinical populations exhibiting integration difficulties.

A general architecture for modeling the dynamics of goal-directed motivation and decision-making.

We present a unified model of the dynamics of goal-directed motivation and decision-making. The model-referred to as the GOAL architecture-provides a quantitative framework for integrating theories of goal pursuit and for relating their predictions to different types of data. The GOAL architecture proposes that motivation changes over time according to three gradients that capture the effects of the distance to the goal (i.e., the progress remaining), the time to the deadline, and the rate of progress required to achieve the goal. This enables the integration and comparison of six theoretical perspectives that make different predictions about how these dynamics unfold when pursuing approach and avoidance goals. Hierarchical Bayesian modeling was used to analyze data from three experiments which manipulate distance to goal, time to deadline, and goal type (approach vs. avoidance), and data from the naturalistic context of professional basketball. The results show that people rely on the distance and rate gradients, and to a lesser degree the time gradient, when making resource allocation decisions during goal pursuit, although the relative influence of the gradients depends on the goal type. We also demonstrate how the GOAL architecture can be used to answer questions about the influence of goal importance. Our findings suggest that goal pursuit unfolds in a complex manner that cannot be accounted for by any one previous theoretical perspective, but that is well-characterized by our unified framework. This research highlights the importance of theoretical integration for understanding motivation and decision-making during goal pursuit. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Sizing up the crowd: Assessing spatial integration difficulties in body size judgements across eating disorder symptomatology.

Introduction: Body size judgements are frequently biased, or inaccurate, and these errors are further exaggerated for individuals with eating disorders. Within the eating disorder literature, it has been suggested that exaggerated errors in body size judgements are due to difficulties with integration. Across two experiments, we developed a novel integration task, named the Ebbinghaus Illusion for Bodies in Virtual Reality (VR), to assess whether nearby bodies influence the perceived size of a single body. VR was used to simulate the appearance of a small crowd around a central target body. Method and Results: In Experiment 1 (N = 412), participants were required to judge the size of a central female target within a crowd. Experiment 1 revealed an Ebbinghaus Illusion, in which a central female appeared larger when surrounded by small distractors, but comparatively smaller when surrounded by large distractors. In other words, the findings of Experiment 1 demonstrate that surrounding crowd information is integrated when judging an individual's body size; a novel measure of spatial integration (i.e., an Ebbinghaus Illusion for Bodies in VR). In Experiment 2 (N = 96), female participants were selected based on high (n = 43) and low (n = 53) eating disorder symptomatology. We examined whether the magnitude of this illusion would differ amongst those with elevated versus low eating disorder symptomatology, in accordance with weak central coherence theory, with the high symptomatology group displaying less spatial integration relative to the low group. The results of Experiment 2 similarly found an Ebbinghaus Illusion for Bodies in VR. However, illusion magnitude did not vary across high and low symptomatology groups. Discussion: Overall, these findings demonstrate that surrounding crowd information is integrated when judging individual body size; however, those with elevated eating disorder symptomatology did not show any integration deficit on this broader measure of spatial integration.

How do recall requirements affect decision-making in free recall initiation? A linear ballistic accumulator approach.

Models of free recall describe free recall initiation as a decision-making process in which items compete to be retrieved. Recently, Osth and Farrell (Psychological Review, 126, 578-609, 2019) applied evidence accumulation models to complete RT distributions and serial positions of participants' first recalls in free recall, which resulted in some novel conclusions about primacy and recency effects. Specifically, the results of the modeling favored an account in which primacy was due to reinstatement of the start-of-the-list, and recency was found to be exponential in shape. In this work, we examine what happens when participants are given alternative recall instructions. Prior work has demonstrated weaker primacy and greater recency when fewer items are required to report (Ward & Tan, Memory & Cognition, 2019), and a key question is whether this change in instructions qualitatively changes the nature of the recall process, or merely changes the parameters of the recall competition. We conducted an experiment where participants studied six- or 12-item lists and were post-cued as to whether to retrieve a single item, or as many items as possible. Subsequently, we applied LBA models with various assumptions about primacy and recency, implemented using hierarchical Bayesian techniques. While greater recency was observed when only one item was required for output, the model selection did not suggest that there were qualitative differences between the two conditions. Specifically, start-of-list reinstatement and exponential recency functions were favored in both conditions.

Assessing the Efficacy and Acceptability of a Web-Based Intervention for Resilience Among College Students: Pilot Randomized Controlled Trial.

BACKGROUND: College students are at elevated risk for developing mental health problems and face specific barriers around accessing evidence-based treatment. Web-based interventions that focus on mental health promotion and strengthening resilience represent one possible solution. Providing support to users has shown to reduce dropout in these interventions. Further research is needed to assess the efficacy and acceptability of these interventions and explore the viability of automating support. OBJECTIVE: This study investigated the feasibility of a new web-based resilience program based on positive psychology, provided with human or automated support, in a sample of college students. METHODS: A 3-armed closed pilot randomized controlled trial design was used. Participants were randomized to the intervention with human support (n=29), intervention with automated support (n=26), or waiting list (n=28) group. Primary outcomes were resilience and well-being, respectively measured by the Connor-Davidson Resilience Scale and Pemberton Happiness Index. Secondary outcomes included measures of depression and anxiety, self-esteem, and stress. Outcomes were self-assessed through online questionnaires. Intention-to-treat and per-protocol analyses were conducted. RESULTS: All participants demonstrated significant improvements in resilience and related outcomes, including an unexpected improvement in the waiting list group. Within- and between-group effect sizes ranged from small to moderate and within-group effects were typically larger for the human than automated support group. A total of 36 participants began the program and completed 46.46% of it on average. Participants were generally satisfied with the program and found it easy to use. CONCLUSIONS: Findings support the feasibility of the intervention. Preliminary evidence for the equal benefit of human and automated support needs to be supported by further research with a larger sample. Results of this study will inform the development of a full-scale trial, from which stronger conclusions may be drawn. TRIAL REGISTRATION: International Standard Randomized Controlled Trial Number (ISRCTN) 11866034; http://www.isrctn.com/ISRCTN11866034. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.1016/j.invent.2019.100254.

Prospective Memory Performance in Simulated Air Traffic Control : Robust to Interruptions but Impaired by Retention Interval.

OBJECTIVE: To examine the effects of interruptions and retention interval on prospective memory for deferred tasks in simulated air traffic control. BACKGROUND: In many safety-critical environments, operators need to remember to perform a deferred task, which requires prospective memory. Laboratory experiments suggest that extended prospective memory retention intervals, and interruptions in those retention intervals, could impair prospective memory performance. METHOD: Participants managed a simulated air traffic control sector. Participants were sometimes instructed to perform a deferred handoff task, requiring them to deviate from a routine procedure. We manipulated whether an interruption occurred during the prospective memory retention interval or not, the length of the retention interval (37-117 s), and the temporal proximity of the interruption to deferred task encoding and execution. We also measured performance on ongoing tasks. RESULTS: Increasing retention intervals (37-117 s) decreased the probability of remembering to perform the deferred task. Costs to ongoing conflict detection accuracy and routine handoff speed were observed when a prospective memory intention had to be maintained. Interruptions did not affect individuals' speed or accuracy on the deferred task. CONCLUSION: Longer retention intervals increase risk of prospective memory error and of ongoing task performance being impaired by cognitive load; however, prospective memory can be robust to effects of interruptions when the task environment provides cuing and offloading. APPLICATION: To support operators in performing complex and dynamic tasks, prospective memory demands should be reduced, and the retention interval of deferred tasks should be kept as short as possible.

Author Correction: A consensus-based transparency checklist.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Using response time distributions and race models to characterize primacy and recency effects in free recall initiation.

Primacy and recency effects are common benchmarks for models of free recall and episodic memory. In this work, we show that RT distributions carry diagnostic information about how items enter into competition for recall, and how that competition impacts on the dynamics of recall and leads to novel conclusions about the forms of primacy and recency effects. We jointly fit RT distributions and serial position functions for free recall initiation with both a racing diffusion model and the linear ballistic accumulator (LBA: Brown & Heathcote, 2008). The models were fit in a hierarchical Bayesian framework, factorially varying different assumptions of how primacy and recency are generated. Recency functions were either exponential or power law in shape. Primacy was treated either as a strength boost to the early list items so that both primacy and recency items jointly compete to be retrieved; a mixture of primacy and recency gradients reflecting the usage of different retrieval cues; or a primacy-as-recency account in which primacy items are functionally recent due to the contribution of rehearsal. Although serial position curves do not distinguish between these accounts, they make distinct predictions about how RT distributions vary across serial positions. Results from a number of data sets strongly favor an exponential recency function along with a mixture model of primacy and recency gradients. These results suggest that complete RT distributions can provide informative constraints on models of free recall. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Analyzing social media data: A mixed-methods framework combining computational and qualitative text analysis.

To qualitative researchers, social media offers a novel opportunity to harvest a massive and diverse range of content without the need for intrusive or intensive data collection procedures. However, performing a qualitative analysis across a massive social media data set is cumbersome and impractical. Instead, researchers often extract a subset of content to analyze, but a framework to facilitate this process is currently lacking. We present a four-phased framework for improving this extraction process, which blends the capacities of data science techniques to compress large data sets into smaller spaces, with the capabilities of qualitative analysis to address research questions. We demonstrate this framework by investigating the topics of Australian Twitter commentary on climate change, using quantitative (non-negative matrix inter-joint factorization; topic alignment) and qualitative (thematic analysis) techniques. Our approach is useful for researchers seeking to perform qualitative analyses of social media, or researchers wanting to supplement their quantitative work with a qualitative analysis of broader social context and meaning.

Anticipatory access to group-level information in working memory.

One assumption common to many serial recall models is that lists can be stored in a structured manner, as groups nested inside larger sequences. However, many of these theories fail to explain the dynamics by which those groups are accessed, and those models that do provide such an account have not been comprehensively tested. This article presents three experiments using a probed recall paradigm in which potential target positions were precued. Across all the experiments, it was found that a precue that targeted only a single group led to faster recall than one that targeted multiple groups, in cases where the number of positions precued was equated. In two of the three experiments, there was evidence (quantified using Bayes factors [BFs]) that responding under a precue targeting multiple groups was no faster than an uniformative control precue. The results suggest that retrieval of groups is a major bottleneck to recall items in structured lists and are consistent with models assuming that information about groups is retrieved prior to accessing elements in the groups themselves.

Benchmarks for models of short-term and working memory.

Any mature field of research in psychology-such as short-term/working memory-is characterized by a wealth of empirical findings. It is currently unrealistic to expect a theory to explain them all; theorists must satisfice with explaining a subset of findings. The aim of the present article is to make the choice of that subset less arbitrary and idiosyncratic than is current practice. We propose criteria for identifying benchmark findings that every theory in a field should be able to explain: Benchmarks should be reproducible, generalize across materials and methodological variations, and be theoretically informative. We propose a set of benchmarks for theories and computational models of short-term and working memory. The benchmarks are described in as theory-neutral a way as possible, so that they can serve as empirical common ground for competing theoretical approaches. Benchmarks are rated on three levels according to their priority for explanation. Selection and ratings of the benchmarks is based on consensus among the authors, who jointly represent a broad range of theoretical perspectives on working memory, and they are supported by a survey among other experts on working memory. The article is accompanied by a web page providing an open forum for discussion and for submitting proposals for new benchmarks; and a repository for reference data sets for each benchmark. (PsycINFO Database Record

Benchmarks provide common ground for model development: Reply to Logie (2018) and Vandierendonck (2018).

We respond to the comments of Logie and Vandierendonck to our article proposing benchmark findings for evaluating theories and models of short-term and working memory. The response focuses on the two main points of criticism: (a) Logie and Vandierendonck argue that the scope of the set of benchmarks is too narrow. We explain why findings on how working memory is used in complex cognition, findings on executive functions, and findings from neuropsychological case studies are currently not included in the benchmarks, and why findings with visual and spatial materials are less prevalent among them. (b) The critics question the usefulness of the benchmarks and their ratings for advancing theory development. We explain why selecting and rating benchmarks is important and justifiable, and acknowledge that the present selection and rating decisions are in need of continuous updating. The usefulness of the benchmarks of all ratings is also enhanced by our concomitant online posting of data for many of these benchmarks. (PsycINFO Database Record